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Safety device for a vehicle 



Introduction 

The present invention relates to a safety device for a vehicle, in particular a 
device for detecting and/or locating objects or persons in the vicinity of a 
vehicle. Such a device may be used, for example, in a passenger restraint 
system for controlling one or more airbags, in a comfort system (for 
automatically setting headrests and/or vehicle seats and/or rear view mirrors 
etc.) or in a system for avoiding accidents. The invention also relates to a 
method for detecting and/or locating objects and persons. 

In order to reduce the risk of injury for the occupants of a vehicle in the event of 
an accident, modern vehicles are generally equipped with an active passenger 
restraint system. Such an active passenger restraint system comprises one or 
more airbags which are inflated instantaneously when the vehicle experiences 
an impact, and which absorb energy which is released by the passenger when 
the impact occurs, and if appropriate seatbelt tensioning devices or the like. 

Since an airbag such as a front-mounted airbag emerges from the dashboard or 
the impact absorber of the steering wheel at very high speeds (200 to 
300 km/h), the passenger can possibly suffer injury from the triggering airbag if 
his sitting position is unfavorable under the circumstances. In order to avoid 
this, microprocessor-controlled restraint systems with a plurality of operating 
modes have been developed which permit various triggering parameters of the 
restraint system to be adapted as a function, inter alia, of the position of the 
passenger on the vehicle seat or his stature, said parameters being, for 
example, the time at which inflation occurs, the inflating speed and/or the 
inflation volume of the airbag. 

In order to enable such adaptation of the triggering parameters of the restraint 
system by means of the microprocessor, the latter must of course be supplied 
with various parameters relating to the sitting position or the stature of the 
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respective passenger. Such a passenger restraint system must consequently 
be equipped with a device for determining these parameters. 

Known devices for detecting the stature or the sitting position of a passenger 
are based, for example, on detecting the weight distribution of the passenger in 
the seat. Based on the determined weight distribution it is possible to draw 
conclusions about the parameters to be detected, using statistical correlation 
functions. 

A completely different approach to detecting a seat occupation is adopted by 
systems which determine a presence and, if appropriate, a sitting position of a 
passenger by means of image-forming systems. In such systems, the 
passenger compartment of the vehicle is monitored by means of one or more 
cameras and the recorded video signal is evaluated by means of suitable image 
processing methods. By using modern CCD or CMOS cameras with high 
resolution it is possible to detect objects relatively accurately by means of such 
systems, i.e. it is possible to distinguish, with relative accuracy, whether the 
vehicle seat is unoccupied or whether it is occupied by a child's seat or by a 
passenger. Furthermore, by detecting the vertical position of the head of a 
passenger it is possible to determine the stature of the passenger. 

In order to detect the sitting position of a passenger with accuracy, there is, in 
addition to the two-dimensional image information, also a need for depth 
information, i.e. information about the distance between the camera and the 
object of interest (for example the head of the passenger). For this purpose, for 
example stereoscopic methods are used which calculate depth information from 
two images which are taken in a laterally offset fashion. A device, which 
operates according to such a method, is known, for example, from 
EP-A-0 885 782. 

As an alternative to. these stereoscopic methods, methods are known which 
evaluate unsharpness of the recorded image with respect to a focal plane v The 
passenger compartment of the vehicle is recorded with a single camera with a 
rather low depth of focus. An object which is located in a plane which is different 
from the focal plane is blurred in the recorded image which . This lack of focus 
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can be detected computationally by means of suitable image processing 
methods and the distance of the object from the focal plane can be determined 
therefrom. Such a method is described, for example, in US-A-6, 198,998. 

The described methods and devices provide good results in object detection by 
5 virtue of the good resolution of modern CCD or CMOS cameras. However, the 
quantity of data which is to be processed in such systems is relatively large, in 
particular when stereoscopic methods are used. Since the position of a 
passenger has to be detected in real time, these systems make enormous 
requirements of the image-processing system. 

10 In addition to these devices which are based on the evaluation of two- 
dimensional recorded images, systems are known which directly supply depth 
information for the image information by measuring the time of flight of 
modulated infrared light. Such 3-D cameras are known, for example, from the 
publications WO-A-96/1 5626 and WO-A-98/10255. In contrast to the 2-D 

15 cameras described above, 3-D cameras have a comparatively low resolution, 
as result of which the quantity of data which is to be processed is significantly 
reduced. Moreover, the depth information which is required to determine the 
position of the object of interest is directly available. However, the low resolution 
of a 3-D camera causes problems for the reliable detection of objects. 

Objective of the invention 

20 The objective of the present invention is consequently to propose an improved 
system for detecting and locating objects. 

General description of the invention 

This objective is achieved according to the invention by means of a device for 
detecting and locating objects comprising 

a 3-D camera for recording three r dimensional situation images of a space 
25 which is to be monitored; 
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first means for evaluating the three-dimensional situation images by means 
of suitable object detection methods for determining an object which is 
present in the space which is to be monitored, and for determining the 
position of the object in the space; 

a 2-D camera for recording two-dimensional situation images of that region 
of the space which is to be monitored, in which the position of the object 
has been determined; and 

second means for evaluating the two-dimensional situation images by 
means of suitable object detection methods for redetermining the object 
which is present in the space which is to be monitored. 

In addition to the device, the invention also relates to a method for detecting 
and locating objects, comprising the steps of: 

recording a first, three-dimensional situation image of a space which is to be 
monitored by means of a 3-D camera, 

evaluating the first, three-dimensional situation image by means of suitable 
object detection methods for determining an object which is present in the 
space which is to be monitored, and for determining the position of the 
object in the space; 

recording a second, two-dimensional situation image of a specific region of 
the space which is to be monitored by means of a 2-D camera, the specific 
region corresponding to the region of the space in which the position of the 
object has been determined; 

evaluating the second, two-dimensional situation image in order to 
redetermine the object which is present. 

Alternatively, the objective is also achieved by means of a device for detecting 
and locating objects comprising 

a 3-D camera for recording three-dimensional situation images of a space 
which is to be monitored; 
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first means for evaluating the three-dimensional situation images by means 
of suitable object detection methods for determining an object which is 
present in the space which is to be monitored, and for determining the 
position of the object in the space; 

a 2-D camera for recording two-dimensional situation images of the space 
which is to be monitored; and 

second means for evaluating a specific region of the two-dimensional 
situation images by means of suitable object detection methods for 
redeterming the object which is present in the space which is to be 
monitored, the specific region corresponding to the region of the space in 
which the position of the object has been determined. 

In this case, the method for detecting and locating objects comprises method for 
detecting and locating objects the steps of: 

recording a first, three-dimensional situation image of a space which is to be 
monitored by means of a 3-D camera, 

evaluating the first, three-dimensional situation image by means of suitable 
object detection methods for determining an object which is present in the 
space which is to be monitored, and for determining the position of the 
object in the space; 

recording a second, two-dimensional situation image of the space which is 
to be monitored by means of a 2-D camera; 

evaluating a specific region of the second, two-dimensional situation image 
is evaluated in order to determine once more the object which is present, 
the specific region corresponding to the region of the space in which the 
position of the object has been determined. 

The device according to the invention and the method according to the 
invention advantageously combine the advantages in terms of high speed and 
the small quantity of data of a 3-D camera system with the high resolution and 
the resulting reliable detection of the object of 2-D camera systems. In fact, with 
the proposed device, the entire space which is to be monitored is firstly sensed 
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using a 3-D camera and the image which is acquired in the process is evaluated 
in view of a first object detection. Since the resolution of a 3-D camera is 
relatively low, the quantities of data which have to be processed are 
correspondingly small. The depth information which is obtained at the same 
time can be used directly for determining positions. 

Then, using the 2-D camera, only the region of the space to be monitored which 
is of interest, i.e. the region in which the position of the object has been 
determined, can be sensed once more using the 2-D camera. The high- 
resolution two-dimensional image, which is acquired in this context, is then 
processed for a new object detection. In the alternative described embodiment 
of the invention, a two-dimensional image of the entire region which is to be 
monitored is taken using the 2-D camera but only the region of the image which 
is of interest is evaluated. 

The first object detection, which has been carried out on the basis of a three- 
dimensional image, can be verified by means of this new evaluation, as a result 
of which the disadvantage of the low resolution of the three-dimensional image 
can be compensated for. It is to be noted that the quantity of data which is to be 
processed when verifying the object detection is significantly reduced in 
comparison with a conventional object detecting process in the space since, 
during this verification process, either only the region of the space which is of 
interest is sensed and processed or only the region of the 2-D image which is of 
interest is evaluated. In comparison to the conventional object detection 
methods, both alternatives of the invention accordingly reduce (in different 
ways) the quantity of data which is to be processed for the 2-D object detection. 

The method according to the invention is accordingly characterized by an 
extremely fast detection of the position of an object and a high degree of 
reliability of the object detection while at the same time the quantities of data to 
be processed are small. 

It is to be noted that in the first embodiment of the invention the 2-D image 
region, which is recorded, can be restricted to the region of interest either by 
swiveling the 2-D camera or by means of a zoom function of the camera. In 
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contrast to this, in the second embodiment, the 2-D camera does not need to be 
of movable or to be equipped with a zoom function. In fact, the region which the 
2-D camera records can correspond to that of the 3-D camera since the 
reduction of the quantity of data to be processed can be carried out by software. 
Only that part of the recorded overall image which contains the object to be 
detected is processed. 

Owing to the small quantities of data, the requirements for the processor power 
of the system are correspondingly low. This preferably permits the first means 
for evaluating the three-dimensional situation images and the second means for 
evaluating the two-dimensional situation images to be implemented in a 
common processor unit. 

In one advantageous embodiment of the device, the 3-D camera and the 2-D 
camera are arranged directly adjacent to one another. The 3-D camera and the 
2-D camera can be arranged, for example, in a common housing. By means of 
such an arrangement, parallax errors between the recorded two-dimensional 
and three-dimensional images can be effectively avoided. Furthermore, the 
assignment of the individual detection regions is made easier. 

In one advantageous embodiment of the invention, the 3-D camera and the 2-D 
camera operate in the same spectral region. Both cameras thereby preferably 
operate with a common light source for illuminating the space which is to be 
monitored with light of a wavelength which is contained in the spectral region, 
for example an infrared region. As a result, errors resulting from different 
illumination of the object which is to be detected are effectively avoided. 
Furthermore, the influence of extraneous illumination can be significantly 
reduced so that simplified algorithms for image processing can be used. 

The light source is advantageously arranged together with the 3-D camera and 
the 2-D camera in a common housing. 

In one application of the method described above, the invention also relates to a 
method for controlling a restraint system in a vehicle, comprising the steps of: 

determining an object and detecting the position of the object in the region 
of a vehicle seat, 
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generating a control signal which is specific to the type and position of the 
object which has been determined and transmitting the control signal to a 
control unit of the restraint system; 

selecting a suitable function mode of the restraint system on the basis of the 
control signal which has been transmitted. 

The invention is used in this case to classify a seat occupation, the high 
detection reliability of the object which is to be detected, for example of a child's 
seat or of a head of a vehicle occupant, enabling a reliable classification of the 
seat occupation. A restraint system which operates according to this method 
can consequently be switched, with a particularly low error quotient, into a mode 
which is appropriate for the occupation situation which is detected at a given 
time. 

It is to be noted that the determination of the position of the head of a vehicle 
occupant permits the sitting height of this occupant to be calculated. One 
advantageous embodiment of the method takes into account this height of the 
torso in the classification of the seat occupation. In this case, the generation of 
a control signal comprises the calculation of a height of a torso of the vehicle 
occupant on the basis of the position of the head. 

The present invention can preferably also be used in a method for avoiding 
accidents. Such a method for avoiding accidents in a vehicle comprises, for 
example, the steps of: determining an object and detecting the position of the 
object in the region of an area surrounding the driver according to the method 
described above, and initiating suitable measures if the specific position of the 
object potentially counteracts safe driving of the vehicle. 

This method can be used, for example, to detect a body posture of the driver 
which is atypical for driving the vehicle. Such an atypical body posture can 
result from acute illness, excessive fatigue or irresponsible behavior of the 
driver. 

In one possible embodiment of the method, the object comprises a body part of 
the driver, the suitable measures being initiated if the position of the body part is 
determined in a region, which is associated with an inappropriate body posture 
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for driving the vehicle. The body part can comprise, for example, the head of the 
driver. If the driver assumes a body posture with his torso inclined forward 
owing to illness or owing to excessive fatigue, the head is in a position, which is 
not suitable for safely driving the vehicle. When such an atypical head position 
is detected, suitable countermeasures can be initiated according to the present 
method. 

A similar atypical head position can also arise due to irresponsible behavior of 
the driver if the driver is, for example, picking up an object from the floor of the 
vehicle or from a rear seat of the vehicle. Such a position of the head can also 
be detected with the present method and corresponding measures can be 
initiated. Alternatively to the head, other body parts of the driver can also be 
monitored for atypical or irresponsible behavior. Thus, for example, the 
detection of a foot in the region of the dashboard can be recognized as an 
extremely dangerous and highly irresponsible body posture. With the aforesaid 
method it is also possible to detect if the driver takes both hands off the steering 
wheel. In all these cases, suitable measures can be initiated such as, for 
example, the generation of a visual or audible warning signal and/or the 
triggering of an emergency brake system and/or the recording of the driver's 
behavior and/or the selection of a suitable function mode of the restraint 
system. It is to be noted here that in particular the recording of the driver's 
behavior can provide important information about the culpability of the driver if 
an accident should occur. 

In another embodiment of the method the object comprises an item, one or 
more of the described measures being initiated if the position of the item is 
determined in a field of vision of the driver. With such a method it is possible, for 
example, to generate an audible warning tone if a highly irresponsible driver is 
studying a map while driving or reading a newspaper, for example in stop-go 
traffic. 

It is to be noted that for the method described above the camera system is 
preferably arranged in a region directly in front of the driver of the vehicle. 
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The method according to the invention and the device according to the 
invention have a further advantageous application in the control of a precrash 
system, wherein suitable safety measures are initiated in accordance with the 
type of the object which is determined and its position in the area outside the 
vehicle. Such a system may, for example, trigger an emergency brake system 
when the vehicle approaches an obstacle in the front region (or in the rear 
region when reversing). In another variant, when a pedestrian is detected in the 
direct vicinity in front of the driving vehicle, an external airbag may be fired or 
the hood of the engine of the vehicle may be tipped into a slightly raised 
position in order to avoid, or at least reduce, injuries to the pedestrian. 

In a further application, the invention can be used in the control of an anti-theft 
warning system in a vehicle. Use is also possible in a comfort-increasing 
system in which various comfort functions are set to the sitting height of the 
respective passenger. 

It is to be noted that, when a plurality of applications are combined in the 
vehicle, a plurality of camera units can be connected to a common electronic 
evaluation system. For example, a first camera unit can monitor the passenger 
compartment of the vehicle in order to detect the seat occupation while a 
second camera unit monitors the outside area in front of the vehicle, and a third 
unit monitors the outside area behind the vehicle. 

It is also to be noted that the existing camera system can also be used in other 
applications. For example, videophone transmission by means of the built-in 
camera is conceivable. 

Description with reference to the figures 

In the following text an embodiment of the invention is described with reference 
to the appended figures, of which: 

Fig. 1 shows a schematic view of an application of the method according to 
the invention for monitoring the passenger compartment in a vehicle; 
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Fig. 2 shows a possible design of a embodiment of a device according to 
the invention; 

Fig. 3 is a flowchart of an embodiment of a method for classifying a seat 
occupation; 

Fig. 4 shows a use of the method in a precrash system; 

Fig. 5 shows a plan view of the method illustrated in fig. 4 for illustrating the 
fields of vision; 

Fig. 6 shows a possible built-in variant of an embodiment of a device 
according to the invention in a vehicle, and 

Fig. 7 shows a combined device for monitoring the outside and inside. 

In figure 1, the use of a device according to the invention for monitoring the 
passenger compartment of a vehicle is illustrated. In such a use, a device 10 for 
detecting and locating an object is arranged, for example, in the front region of 
the vehicle, directly underneath the roof of the vehicle. The illustrated 
embodiment is a camera module for monitoring the front seat passenger's side, 
i.e. the field of vision of the camera is directed toward the front seat passenger's 
side. Alternatively, the device can of course also be arranged in such a way that 
the field of vision of the camera module is directed toward the driver's side. 
Depending on the arrangement and configuration of the camera module, both 
sides of the vehicle and/or the rear seat region of the vehicle can also be 
monitored simultaneously. 

A possible design of a camera module 10 is illustrated in fig. 2. It comprises 
essentially a 3-D camera 12 with connected electronics 14, a 2-D camera 16 
with connected electronics 18 and a suitable illumination source 20 with 
associated driver 22. The cameras are preferably arranged directly adjacently to 
one another so that parallax errors between the two-dimensional and the three- 
dimensional images are avoided. The 3-D camera, the 2-D camera 16 and the 
illumination source 20 are preferably arranged in a common housing 24 in 
which the processor unit 26 is also arranged. The module 10 may be 
manufactured as a prefabricated unit which is connected to the control 




P-IEE-085 1 2 

electronics of a restraint system via a specific interface 28 when it is installed in 
the vehicle. 

Fig. 3 shows a flowchart of an embodiment of the method according to the 
invention in the use for controlling an airbag. At first, a three-dimensional image 
5 is taken by means of the 3-D camera and evaluated 100. If this object-detecting 
operation has the result that the object is a head of a passenger 120 (after 
verification of the detection with the 2-D system 140), firstly the height of the 
torso is calculated 160 in order to form a classification 180 of the passenger, 
and on the other hand the position of the head is followed 200, in order to 
10 determine 220 the sitting position. 

If the detected object is not a head of a passenger but rather a child's seat 240, 
the seat occupation is classified as a child's seat occupation after verification 
with the 2-D system 260, and the airbag is correspondingly switched into a 
mode which is suitable for this. 

15 Fig. 4 and fig. 5 show the use of a device in the control of a precrash system. 
Camera modules which are installed at suitable locations monitor the region in 
front of and behind a vehicle in order to initiate corresponding protective 
measures if the vehicle approaches an obstacle. Monitoring of the region to the 
side of the vehicle is also represented. 

20 Figures 6 and 7 illustrate a combined module for sensing the outside and inside 
as well as a possible built-in variant in a vehicle. In the combined system, 
various groups composed of the 3-D camera, 2-D camera and illumination 
system are connected to a single evaluation unit. This can permit a 
considerable saving of space. 
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Patent Claims 

1 . A device for detecting and locating objects comprising 

a 3-D camera for recording three-dimensional situation images of a space 
which is to be monitored; 

first means for evaluating the three-dimensional situation images by means 
of suitable object detection methods for determining an object which is 
present in the space which is to be monitored, and for determining the 
position of the object in the space; 

a 2-D camera for recording two-dimensional situation images of that region 
of the space which is to be monitored, in which the position of the object 
has been determined; and 

second means for evaluating the two-dimensional situation images by 
means of suitable object detection methods for redetermining the object 
which is present in the space which is to be monitored. 

2. A device for detecting and locating objects comprising 

a 3-D camera for recording three-dimensional situation images of a space 
which is to be monitored; 

first means for evaluating the three-dimensional situation images by means 
of suitable object detection methods for determining an object which is 
present in the space which is to be monitored, and for determining the 
position of the object in the space; 

a 2-D camera for recording two-dimensional situation images of the space 
which is to be monitored; and 

second means for evaluating a specific region of the two-dimensional 
situation images by means of suitable object detection methods for 
redetermining the object which is present in the space which is to be 
monitored, the specific region corresponding to the region of the space in 
which the position of the object has been determined. 
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3. The device as claimed in claim 1 or 2, wherein the first means for evaluating 
the three-dimensional situation images and the second means for 
evaluating the two-dimensional situation images are implemented in a 
common processor unit. 

4. The device as claimed in one of claims 1 to 3, wherein the 3-D camera and 
the 2-D camera are arranged directly adjacent to one another. 

5. The device as claimed in claim 4, wherein the 3-D camera and the 2-D 
camera are arranged in a common housing. 

6. The device as claimed in one of claims 1 to 5, wherein the 3-D camera and 
the 2-D camera operate in the same spectral region. 

7. The device as claimed in claim 6, comprising a light source for illuminating 
the space which is to be monitored with light of a wavelength which is 
contained in the spectral region. 

8. The device as claimed in one of claims 6 or 7, wherein the spectral region 
comprises an infrared region. 

9. The device as claimed in claim 6, wherein the 3-D camera, the 2-D camera 
and the light source are arranged in a common housing. 

10. A method for detecting and locating objects, comprising the steps: 

a) recording a first, three-dimensional situation image of a space which is 
to be monitored by means of a 3-D camera, 

b) evaluating the first, three-dimensional situation image by means of 
suitable object detection methods for determining an object which is 
present in the space which is to be monitored, and for determining the 
position of the object in the space; 

c) recording a second, two-dimensional situation image of a specific 
region of the space which is to be monitored by means of a 2-D 
camera, the specific region corresponding to the region of the space in 
which the position of the object has been determined; 
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d) evaluating the second, two-dimensional situation image in order to 
redetermine the object which is present. 

11. A method for detecting and locating objects, comprising the steps 

a) recording a first, three-dimensional situation image of a space which is 
to be monitored by means of a 3-D camera, 

b) evaluating the first, three-dimensional situation image by means of 
suitable object detection methods for determining an object which is 
present in the space which is to be monitored, and for determining the 
position of the object in the space; 

c) recording a second, two-dimensional situation image of the space which 
is to be monitored by means of a 2-D camera; 

d) evaluating a specific region of the second, two-dimensional situation 
image in order to redetermine the object which is present, the specific 
region corresponding to the region of the space in which the position of 
the object has been determined. 

12. The method as claimed in claim 10 or 11, wherein the 3-D camera and the 
2-D camera operate in the same spectral region. 

13. The method as claimed in claim 12, comprising the step of illuminating the 
space which is to be monitored with light of a wavelength which is contained 
in the spectral region. 

14. The method as claimed in one of claims 12 or 13, wherein the spectral 
region comprises an infrared region. 

15. A method for controlling a restraint system in a vehicle, comprising the 
steps: 

a) determining an object and the position of said object in the region of a 
vehicle seat according to a method as claimed in one of claims 10 to 
14, 
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b) generating a control signal which is specific to the type and position of 
the object which has been determined and transmitting the control 
signal to a control unit of the restraint system; 

c) selecting a suitable function mode of the restraint system on the basis 
of the control signal which has been transmitted. 

16. The method as claimed in claim 15, wherein the object comprises a child's 
seat. 

17. The method as claimed in claim 15, wherein the object comprises the head 
of a vehicle occupant. 

18. The method as claimed in claim 17, wherein the generation of a control 
signal comprises the calculation of a height of a torso of the vehicle 
occupant on the basis of the position of the head. 

19. A method for avoiding accidents in a vehicle, comprising the steps: 

a) determining an object and the position of said object in the region of an 
area surrounding the driver according to a method as claimed in one of 
claims 10 to 14, 

b) initiating suitable measures if the determined position of the object 
potentially counteracts safe driving of the vehicle. 

20. The method as claimed in claim 19, wherein the object comprises a body 
part of the driver and wherein the suitable measures are initiated if the 
position of the body part is determined in a region which is associated with 
an inappropriate body posture for driving the vehicle. 

21. The method as claimed in claim 19, wherein the object comprises an item, 
and wherein the suitable measures are initiated if the position of the item is 
determined in a field of vision of the driver, 

22. The method as claimed in one of claims 19 to 21, wherein the initiation of 
corresponding measures comprises the generation of a visual or audible 
warning signal and/or the triggering of an emergency brake system and/or 
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the recording of the driver's behavior and/or the selection of a suitable 
function mode of the restraint system. 

23. A use of a device as claimed in one of claims 1 to 9 in the control a restraint 
system in a vehicle, wherein the restraint system is switched to a suitable 
function mode in accordance with the type of the object which is determined 
and its position in the passenger compartment of the vehicle. 

24. The use of a device as claimed in one of claims 1 to 9 in the control a 
precrash system, wherein suitable safety measures are initiated in 
accordance with the type of the object which is determined and its position 
in the area outside the vehicle. 

25. The use of a device as claimed in one of claims 1 to 9 in the control of an 
anti-theft warning system in a vehicle. 
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